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Abstract. This paper is about the learners’ satisfaction comparison between text 
and speech Dialogue-Based Computer-Assisted Language Learning systems (DB- 
CALL system). A DB-CALL system aims to allow learners to talk to the system in a 
foreign language as if they were talking with a native speaker and to provide learning 
feedback that will grammatically correct spoken foreign language or recommend 
a better expression. The satisfaction analysis of the text DB-CALL system was 
conducted by 20 learners. The satisfaction analysis of the speech DB-CALL system 
was conducted by 36 learners. The average satisfaction of the text DB-CALL system 
was 3.44 points. This system needs to improve the ability to respond appropriately to 
out-of-topic conversations (2.10 points). The average satisfaction of the speech DB- 
CALL system was 3.87 points. This system needs to enhance the speech recognition 
function (3.36 points). 


Keywords: satisfaction analysis, computer assisted language learning system, 
dialogue system, DB-CALL. 


1. Introduction 


The DB-CALL system aims to either grammatically correct learners’ foreign 
language utterance or to provide learning feedback that recommends better 
expressions. The Electronics and Telecommunications Research Institute (ETRI) 
has developed the DB-CALL system since 2010. The DB-CALL system developed 
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from 2010 to 2015 was text-based and system-oriented, which restricted the 
learner’s dialogue flow and did not allow free conversation beyond the topic (Choi, 
Kwon, Kim, & Lee, 2016). As a hybrid approach between the DB-CALL system 
and a chat bot became active (Dingli & Scerri, 2013), the DB-CALL systems 
evolved from system-oriented to a mixed form between system-oriented and user- 
oriented. The DB-CALL system developed from 2016 to 2018 was speech-based 
and a mix of system-oriented and user-oriented, that allowed free conversation 
outside the topic and feedback on grammar errors without limiting the learner’s 
flow of conversation (Choi, Kwon, & Kim, 2017). In this paper, we describe the 
learners’ satisfaction comparison between the text DB-CALL system and the 
speech DB-CALL system. 


2. Configuration of ETRI’s speech DB-CALL system 


ETRI’s speech DB-CALL system consists of speech recognition, language 
understanding, computer-assisted language learning, dialogue management, 
language generation, and speech synthesis. As described in detail in Choi et al. 
(2017), the speech DB-CALL system’s strategy for user utterance is shown in 
Figure 1. 


Figure 1. Dialogue strategy of ETRI speech DB-CALL system 


Learner’s utterance [necessary ]} optinal’ | 
| 
Language Understanding No 
> Response of chat bot 
Yes 
pon Dialogue Management No 
Yes 


Language Learning No 


in Interactive response 


No 


Language learning 


Provide error feedback 


Yes 


Interactive response 


When a learner makes an utterance, ETRI speech DB-CALL system first confirms 
whether the utterance is topic-oriented. If the utterance is not a topic conversation, 
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ETRI speech DB-CALL system responds with the chat bot and then returns to the 
topic conversation. If the learner’s utterance is topic-oriented, ETRI speech DB- 
CALL system in turn processes suitability of content, grammar error detection, and 
error severity (Choi et al., 2017). 


3. Text DB-CALL system 


3.1. Evaluation 


The evaluation of the text DB-CALL system was conducted by 20 testers, whose 
English level was beginner (ten) and intermediate (ten). Each tester communicated 
with the system literally about the four topics. Topics for evaluation consist of 
purchasing tickets for a New York city tour, ordering food, talking about health 
habits, and thinking about future currency. Participants performed ten missions on 
four topics, each taking an average of 50 minutes per person. The success rate of 
dialogue turns on four topics was 80.86% as follows: 


Success rate of dialogue turn (%) = (number of correct system’s responses 
to learner’s utterance)/(total number of learners’ utterances) x 100. 
3.2. Learners’ satisfaction 
Satisfaction analysis was based on the questionnaire of 20 people who experimented 
with the text DB-CALL system. Five-point Likert scales were assigned to each 
question. One point was ‘very unsatisfied’ and 5 points were ‘very satisfied’. The 


satisfaction results were as follows (Table 1). 


Table 1. Learners’ satisfaction of text DB-CALL system 


Questionnaire Score 


1. Questions about system performance 


1.1. Did the system respond appropriately? 2.90 
1.2. Do you feel free to talk to the system? 2.90 
1.3. Did the system properly point out grammatical errors? 3.15 


1.4. Did the system adequately recommend sentences that would help you learn? 3.60 


1.5. Did the out-of-topic conversations (ex. chat bots) respond appropriately? 2.10 
Average 2.93 
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2. Questions about system function 


2.1. Do you think the ability to correct grammatical 4.40 
errors is necessary for the conversation system? 

2.2. Do you think the ability to recommend sentences 4.55 
is necessary for the conversation system? 

2.3. Do you think conversations outside of the topic 3.60 
(ex. chat bots) are necessary for the conversation system? 

Average 4.18 
3. Questions about system effects 

3.1. After using the system, did the DB-CALL system 3.60 
help you learn English? 

3.2. Do you think this DB-CALL system helps you learn English better 3.55 
than traditional learning methods (such as conversation texts)? 

3.3. Do you think this DB-CALL system will help you 3.60 
motivate your learning? 

3.4. Does this DB-CALL system have fun? 3.35 
Average 3.53 
4. Questions about future services 

4.1. Would you like to use the online education service 3.65 
of the DB-CALL system? 

4.2. Do you want to use the above service even if it is charged? 2.40 
4.3. Are you willing to repeat many times 3.30 


(on average 4-5 times or more) on a topic conversation? 


Average 3.12 


Overall, the satisfaction rate of the text DB-CALL system was 3.44 points out 
of 5 points. A factor that lowers overall satisfaction is the DB-CALL system’s 
inability to respond appropriately to out-of-topic conversations (2.10 points). 
Therefore, it is expected that the learner’s satisfaction with the text DB-CALL 
system will increase if the topic-oriented chat bot system is improved. 


4. Speech DB-CALL system 


4.1. Evaluation 


The speech DB-CALL system was tested in voice on speech phones. Thirty-six 
learners whose English level was beginner (18) and intermediate (18) were divided 
into three groups. Topics to talk with were given differently for each group. Table 2 
shows the progress of satisfaction analysis. 
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Table 2. Evaluation method of speech DB-CALL system 


Group A Group B Group C 
Participants | 12 college students 12 middle school 12 middle school 
(7 male, 5 female) students students 
(male 1, female 11) (male 2, female 10) 
Topic ¢ How can we save How can we save What healthy habit 
our environment? our environment? do you have? 


¢ What is your opinion 
about DIY fashion? 


¢ What do you think of 
Korean food culture? 


¢ Which role on the 
rowing team do 
you think you’ ll 


be good at? 
Evaluation | ¢ Usability in class 
items 
* Content and Services 
¢ Speech recognition 
How to ¢ System introduction and explanation of learning method 
proceed 


¢ Distribution of test equipment 


¢ Pre-learning: after learning words, sentences and expression, 
3 free conversations with DB-CALL system 


¢ Learning 
¢ Confirm result 


¢ Fill out the questionnaire 


4.2. Learners’ satisfaction 


Five-point Likert scales were assigned to each question. One point was ‘very 
poor’ and 5 points were ‘very satisfied’. The satisfaction results were as follows 
(Table 3). 


Satisfaction with speech DB-CALL system was 3.87 out of 5 points. Specifically, 
the satisfaction of usability in class was 4.11, the content and service satisfaction 
was 4.14, and the speech recognition satisfaction was 3.36. The speech recognition 
function should be improved. The satisfaction analysis of the speech DB-CALL 
system is similar to that of the online survey of speech assisted language learning 
of Grimshaw, Cardoso, and Collins (2017). If technology of speech recognition 
improves further, the satisfaction level of Table 3 below will be higher. 
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Table 3. Learners’ satisfaction of speech DB-CALL system 
Group A Group B Group C Total 
Usability 3.92 4.50 3.92 4.11 
in class 
very satisfied(0) very satisfied(6) very satisfied(1) 
satisfied(11) satisfied(6) satisfied(9) 
moderate(1) moderate(0) moderate(2) 
unsatisfied(0) unsatisfied(0) unsatisfied(0) 
very unsatisfied(0) very unsatisfied(0) _ very unsatisfied(0) 
Content and 3.83 4.50 4.08 4.14 
Services 
very satisfied(0) very satisfied(6) very satisfied(2) 
satisfied(10) satisfied(6) satisfied(9) 
moderate(2) moderate(0) moderate(1) 
unsatisfied(0) unsatisfied(0) unsatisfied(0) 
very unsatisfied(0) | very unsatisfied(0) _| very unsatisfied(0) 
Speech 3.00 3.83 3.25 3.36 
recognition 
very satisfied(1) very satisfied(1) very satisfied(1) 
satisfied(2) satisfied(8) satisfied(4) 
moderate(5) moderate(3) moderate(4) 
unsatisfied(4) unsatisfied(0) unsatisfied(3) 
very unsatisfied(0) very unsatisfied(0) _ very unsatisfied(0) 
Total 3.87 


5, Conclusions 


In this paper, we described the learners’ satisfaction comparison between text and 
speech DB-CALL system. The average satisfaction rate of the text DB-CALL 
system was 3.44 points. The average satisfaction rate of the speech DB-CALL 
system was 3.87 points. 


In the future, we will further enhance the performance of the speech DB-CALL 
system by improving speech recognition and dialogue management modules. 
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